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ABSTRACT 


The  Hierarchical  Stepwise  Optimization  algorithm  of  Beaulieu 
and  Goldberg  (1989)  is  applied  to  the  segmentation  of  satellite 
images  into  meaningful,  large-scale  cloud  features.  GOES-W 
visible  imagery  is  used.  Several  different  forms  of  the  cost 
function  are  explored  in  an  attempt  to  improve  the  segmentation 
of  the  Nov.  15,  1983  case.  A  new  cost  function  is  shown  to 
result  in  a  superior  segmentation.  Both  HSWO  versions  are  tested 
on  six  additional  cases.  A  modification  to  the  HSWO  approach  is 
suggested  for  future  research. 
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USE  OF  HIERARCHICAL  STEPWISE  OPTIMIZATION  FOR 
THE  SEGMENTATION  OF  CLOUD  FEATURES 

1 .  Introduction 

In  Peak  (1990a) ,  neural  networks  were  investigated  as  poten¬ 
tial  tools  for  the  automated  classification  of  satellite  cloud 
features.  An  architecture,  called  CIFRS  (Cloud  Image  Feature 
Recognition  System) ,  was  proposed.  CIFRS  would  use  several 
neural  networks  tailored  to  the  interpretation  of  different  types 
of  cloud  features,  as  well  as  statistical  and  knowledge-based 
components  to  augment  and  direct  the  analysis  process. 

The  encouraging  results  of  the  preliminary  experiment  in 
Peak  (1990a)  led  to  an  expanded  study  using  manually-derived 
large-scale  cloud  features  (Peak,  1990b) .  Using  a  series  of  GOES 
images,  a  human  expert  outlined  and  classified  large-scale  cloud 
features  and  identified  the  types  of  clouds  contained  in  them. 
Peak  (1990b)  showed  that  a  neural  network  could  classify  these 
features  with  high  levels  of  accuracy  by  using  only  a  crude  set 
of  inputs. 

Peak  (1990b)  recognized  that  a  portion  of  the  classification 
skill  in  those  experiments  resulted  from  the  fact  that  a  human 
expert  had  already  divided  the  image  into  meaningful  re''^ions  for 
the  neural  network  to  classify.  The  location  and  shape  of  those 
regions  turned  out  to  contain  important  information  that  could  be 
exploited  by  the  neural  networks.  For  this  reason.  Peak  (1990b) 
included  an  initial  experiment  in  image  segmentation  using  the 
Hierarchical  Stepwise  Optimization  (HSWO)  algorithm  of  Beaulieu 
and  Goldberg  (1989) .  A  single  image  was  divided  into  a  set  of 
coarse-resolution,  cloudiness-percentage  "pixels."  The  applica¬ 
tion  of  HSWO  to  this  image  showed  enough  promise  that  an 


expanded  study  of  this  approach  was  recommended  (Peak,  1990b) . 

The  purpose  of  this  paper  is  to  present  the  results  of  this 
expanded  test. 

2.  Digitized  Image  Data 

The  data  used  in  this  study  are  taken  from  the  same  GOES-W 
cases  used  in  Peak  (1990b)  because  the  desired  segmentation  into 
large-scale  cloud  regions  has  already  been  performed  by  Mr.  R. 
Fett  of  NOARL.  The  images  were  digitized  to  75  dots-per-inch 
using  a  Xerox  scanner  at  the  W.R.  Church  Computer  Center  at  the 
U.S.  Naval  Postgraduate  School.  The  regions  digitized  were 
approximately  8  inches  wide  by  4.75  inches  tall,  or  356x600 
pixels,  yielding  nearly  215,000  values  per  image.  Since  the  HSWO 
cost  function  (Peak,  1990b)  must  be  evaluated  for  each  pixel  and 
its  adjacent  neighbors,  the  scheme  begins  with  as  many  cost 
values  as  approximately  4  times  the  number  of  pixels.  Thus,  if 
we  were  to  start  with  the  basic,  digitized  grayshade  values, 

HSWO  would  have  to  calculate,  and  then  find  the  minimum  of, 
860,000  cost  values.  After  combining  the  two  pixels  with  the 
lowest  cost,  there  would  be  about  859,996  cost  values  from  which 
to  find  the  minimum,  and  so  on. 

Obviously,  the  computer  resources  required  for  this  problem 
are  immense  unless  we  reduce  the  amount  of  initial  data.  In 
this  study,  it  was  decided  to  average  the  values  over  regions  of 
10  pixels  X  10  pixels.  Thus,  the  effective  resolution  is 
reduced  to  7.5  dots  per  inch  and  the  region  scanned  becomes 
36x60  pixels;  a  total  of  2160  values  per  image.  This  data  set 
is  much  more  manageable,  although  even  this  size  problem  takes 
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about  2  h  to  complete  on  the  Masscomp  5450. 

A  total  of  seven  images  are  segmented  in  this  study.  The 
first  image  is  the  same  Nov.  15,  1983  image  from  the  preliminary 
test  of  Peak  (1990b).  In  the  next  section,  several  different 
segmentations  of  this  image  will  be  performed  in  an  attempt  to 
determine  the  best  cost  function  to  be  used  for  cloud  imagery. 

The  six  remaining  images  are  those  taken  every  three  days  begin¬ 
ning  with  Oct.  1,  1983  and  ending  with  Oct.  16,  1983.  These 
situations  contain  some  very  complex  cloud  regions  which  should 
provide  a  wide  range  of  segmentation  problems  for  the  scheme. 

3.  Cost  Function  Experiments 

The  HSWO  algorithm  is  straightforward.  For  each  pixel,  a 
region  is  assigned.  The  cost  of  combining  any  region  i  with 

one  of  its  adjacent  regions  j  is  given  by 


Ni  *  Nj 

Ci  j  - -  (x^  -  XO 

■’  +  Nj 


(1) 


where  Nj^  and  Nj  are  the  number  of  pixels,  and  Xj^  and  Xj  are  the 
average  grayshade  values,  of  regions  i  and  j,  respectively.  The 
two  regions  with  the  minimum  cost  are  combined,  and  the  new 
region  is  assigned  the  weighted  average  grayshade  of  the  two 
combined  regions.  The  N-terms  in  (1)  tend  to  assign  the  lowest 
cost  to  combining  two  very  small  regions.  A  small  region  being 
combined  into  a  large  region  is  also  given  a  relatively  low  cost. 
The  highest  cost  results  from  trying  to  combine  two,  large  re¬ 
gions.  The  x-term  in  (1)  accounts  for  the  difference  in  gray¬ 
shade  between  the  regions;  obviously  assigning  lower  costs  when 
the  grayshades  have  similar  values. 
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Before  presenting  the  image  segmentation  results,  the  crite¬ 
rion  for  terminating  the  HSWO  process  should  be  mentioned.  As  in 
Beaulieu  and  Goldberg  (1989)  and  in  Peak  (1990b) ,  the  increase  in 
cost  function  between  iterations  is  used.  First,  we  assume  that, 
for  the  large  scale  of  cloud  regions  we  are  interested  in,  there 
should  be  a  reasonably  small  number  of  such  features  present  in 
the  region  of  interest.  We  arbitrarily  choose  not  to  consider 
stopping  the  process  when  there  are  more  than  20  regions  in  the 
image.  Thus,  even  large  increases  in  the  cost  function  are 
ignored  until  the  algorithm  has  combined  the  pixels  into  20 
separate  regions.  After  that  point,  we  look  for  a  large 
increase  in  cost  between  iterations  to  indicate  the  stopping 
point.  Because  the  minimum  cost  value  sometimes  oscillates  up 
and  down,  there  may  be  an  apparently  large  increase  in  the 
minimum  cost  between  iterations  that  is  mostly  due  to  a  large 
decrease  in  the  previous  iteration  rather  than  a  large  increase 
in  the  present  iteration.  In  other  words,  we  want  to  look  for  a 
large  jump  from  the  minimum  cost  curve  trend  rather  than  from 
individual  values.  The  approach  of  using  an  average  or  weighted 
average  of  previous  minimum  cost  values  as  a  basis  for 
comparison  was  considered.  However,  it  was  found  that  the  cost 
jump  between  the  cost  at  the  current  iteration  and  the  largest 
of  the  costs  at  the  two  previous  iterations  was  sufficient  to 
define  the  stopping  point.  A  jump  of  at  least  40%  is  deemed 
sufficient  in  this  context  (at  least  for  now) . 

In  this  section,  the  focus  is  exclusively  on  the  Nov.  15, 
1983  image  (Fig.  1) .  The  major  features  to  be  isolated  are:  1)  a 
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Figure  1.  Portion  of  GOES-W  image  for  Nov.  15,  1983  used  for 
digitization  and  later  segmentation  using  HSWO. 
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frontal  band  in  the  western,  mid-latitudes,  2)  a  frontal  band 
extending  offshore  from  the  Pacific  northwest,  3)  a  region  of 
stratocumulus  west  of  Baja  California,  and  4)  a  long,  inter- 
tropical  convergence  zone  (ITCZ)  spanning  the  tropics.  There  are 
many  smaller-scale  features  as  well,  but  the  focus  of  this  study 
is  to  attempt  the  segmentation  of  large-scale  cloud  features 
only. 

The  HSWO  segmentation  of  the  Nov.  15  case  using  7.5  dpi 
resolution  is  presented  in  Fig.  2.  All  of  the  image  segmenta¬ 
tions  presented  in  this  paper  are  in  the  form  of  three  figures. 
The  first  is  the  image  to  be  segmented  with  the  segmentation 
regions  overlaid  in  thick,  black  lines  (e.g.,  Fig.  2a).  It  was 
difficult  with  available  photocopiers  to  get  an  overlaid  segmen¬ 
tation  to  show  up  clearly  against  an  image  background.  For  this 
reason,  a  figure  of  the  segmentation  regions  only  (e.g.,  Fig.  2b) 
is  also  presented  below  the  overlaid  image.  The  reader  should  be 
able  to  see  the  segmentation  regions  clearly  in  the  bottom  figure 
and  then  look  for  the  corresponding  regions  in  the  overlaid 
figure  above.  The  areas  in  these  figures  are  labelled  by  the 
algorithm  during  the  segmentation  process.  Thus,  the  same  cloudy 
area  in  one  segmentation  may  have  a  different  label  in  subsequent 
segnentations .  Finally,  the  minimum  cost  function  values  from 
the  last  20  iterations  of  the  segmentation  are  plotted  (e.g.. 

Fig.  2c)  including  an  arrow  indicator  of  the  stopping  point  used. 
The  reader  will  find  that  these  cost  function  curves  show  that 
the  satellite  imagery  segmentations  tend  to  have  the  large  jumps 
in  minimum  cost  suggested  by  Beulieu  and  Goldberg  (1989)  as 
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Figure  2a.  As  in  Fig,  1  except  after  HSWO  segmentation,  HSWO- 
defined  regions  are  outlined  in  heavy,  black  lines  and  labeled 
with  numbers  or  characters. 
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Figure  2c.  Minimum  cost  values  as  a  function  of  the  number  of 
regions  left  uncombined  duriiig  the  segmentation  of  the  image  in 
Fig.  i.  Arrow  indicates  the  cutoff  point  resulting  in  the  seg¬ 
mentation  regions  depicted  in  Fig.  2b. 


stopping  points. 

The  segmentation  of  the  Nov.  15  case  resulted  in  six  regions 
(Fig.  2) .  The  minimum  cost  function  curve  (Fig.  2c)  shows  that 
the  minimum  cost  gradually  increases  until  there  are  seven  re¬ 
gions.  At  this  point,  there  is  a  sharp  decrease,  followed  by  a 
large  increase.  We  choose  the  minimum  value  at  six  regions  as 
the  stopping  point. 

The  image  segmentation  (Fig.  2a  &  b)  shows  that  HSWO  does  an 
excellent  job  of  outlining  the  western  front  (area  E) .  The  main 
body  of  the  Pacific  northwest  front  (area  D)  is  also  segmented 
well.  There  is  a  dim,  warm  front  (that  doesn't  show  up  well  in 
these  Figures) ,  extending  from  Washington  to  northwest  Colorado, 
that  is  included  in  area  D.  One  disadvantage  in  using 
navigated,  photographic  images  is  that  the  white,  coastal  lines 
overlaid  on  the  image  tend  to  contaminate  the  gray-shade  values. 
For  this  reason,  area  D  extends  to  the  northwest  to  include  the 
bright,  complex  coastline  of  Alaska  and  British  Columbia.  A 
major  disappointment  is  the  failure  of  HSWO  to  include  the  thin, 
trailing  frontal  band  into  area  D.  This  band  has  instead  been 
incorporated  into  the  mostly  noncloudy  area  C.  This  region  has 
no  meaning  in  terms  of  cloud  areas;  it  occurs  because  the  plane¬ 
tary  albedo  causes  it  to  have  some  brightness.  Interestingly, 
the  planetary  portion  of  C  has  been  included  with  the  outer 
space  background  areas  at  the  top  left  and  right  of  the  image. 
Area  4  is  a  similar,  noncloudy  planetary  area.  Area  B  indicates 
that  the  ITCZ  is  well-handled  by  HSWO;  it  spans  the  width  of  the 
image  ignoring  local  breaks  and  details  in  the  cloud  region. 
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The  stratocumulus  region  is  outlined  (area  A) ,  but  the  long, 
thin  cloud  band  extending  to  the  north  has  caused  HSWO  to  in¬ 
clude  it  into  a  broad,  dim  area  that  spans  the  planetary  bound¬ 
ary  of  the  image. 

For  the  remainder  of  this  section,  attempts  to  improve  on 
this  segmentation  will  be  presented.  In  particular,  we  want  the 
routine  to  be  able  to  discriminate  the  stratocumulus  region  and 
to  include  the  thin,  trailing  front  into  the  body  of  the  eastern 
frontal  band.  The  latter  problem  may  be  caused  by  the  low  reso¬ 
lution  of  the  data.  In  other  words,  when  the  data  are  reduced  to 
7.5  dpi,  each  "pixel"  along  the  thin  cloud  band  will  include 
significant  noncloudy  information  along  with  the  cloudy  informa¬ 
tion.  When  these  data  are  averaged  to  7.5  dpi,  the  resulting 
grayshade  is  artificially  low,  causing  these  pixels  to  be  includ¬ 
ed  in  the  noncloudy  region  rather  than  into  the  body  of  the 
front. 

To  test  this  hypothesis,  a  new  data  set  was  created  by 
averaging  to  15  dpi  "pixels."  The  hope  is  that  HSWO  will  now 
have  a  thin  line  of  smaller,  but  brighter,  pixels  to  connect  to 
the  body  of  the  front.  The  resulting  HSWO  segmentation  is  pre¬ 
sented  in  Fig.  3.  The  minimum  cost  function  curve  (Fig.  3c) 
indicates  that  the  cutoff  should  again  be  at  six  regions.  The 
resulting  segmentation  (Fig.  3a  &  b)  again  handles  the  western 
front  (area  8)  well.  The  ITCZ  is  now  broken  into  two,  major 
regions  B  and  C.  Unfortunately,  the  eastern  front  (area  9)  still 
does  not  include  the  tailing  portion.  Even  with  the  higher 
resolution,  the  average  grayshade  of  the  tailing  portion  is  still 
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Figure  3a.  As  in  Fig.  2a  except  for  HSWO  segmentation  using 
high-resolution  data  set. 


Figure  3b.  As  in  Fig.  2b  except  for  HSWO  segmentation  using 
high-resolution  data  set. 
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closer  to  that  of  the  dark  background  rather  than  the  bright 
front.  Also,  the  stratocumulus  region  is  now  also  absorbed  into 
the  background  region  A.  It  appears  that  the  problem  lies  in  the 
way  that  the  cost  function  treats  regions  with  intermediate 
grayshades.  Such  regions,  are  susceptible  to  being  joined  into 
adjacent  dark  regions  when  their  grayshade  value  is  even  slightly 
closer  to  being  dark  than  bright. 

Visiting  experts  from  the  National  Academy  of  Sciences 
suggested  that  filtering  the  data  before  HSWO  processing  might 
improve  the  segmentation.  The  idea  is  to  decrease  the  abrupt 
shift  in  brightness  at  region  edges  to  provide  a  more  gradual 
linking  of  adjacent,  cloudy  regions.  The  Nov.  15  case  was  fil¬ 
tered  using  a  simple,  9-point  filter.  The  ensuing  segmentation 
is  presented  in  Fig.  4.  Again,  the  western  front  (area  D)  is 
well  defined.  The  ITCZ  is  now  divided  into  two,  separate  bands 
(areas  A  and  E) .  The  eastern  front  still  does  not  include  the 
trailing  front  and  the  stratocumulus  region  is  also  combined 
into  the  background  area  B.  It  appears  that  the  filtering  has 
not  accomplished  its  desired  effect,  possible  because  filtering 
also  lowers  the  grayshade  of  the  intermediate  regions,  thus 
making  them  more  susceptible  to  inclusion  in  the  darker  areas. 

The  nature  of  the  cloud  regions  we  are  trying  to  segment  is 
that  any  brightness  should  indicate  clouds,  and  that  such  regions 
should  almost  always  be  included  into  adjacent,  cloudy  regions. 
Thus,  even  relatively  dim,  intermediate  regions  should  be  treated 
as  being  more  like  other,  brighter  regions  than  as  dark  regions. 
To  accomplish  this  behavior,  the  cost  function  could  be  modified. 
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Figure  4a.  As  in  Fig.  2a  except  for  HSWO  segmentation  using 
filtered  data  set. 


Figure  4b.  As  in  Fig.  2b  except  for  HSWO  segmentation  using 
filtered  data  set. 
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We  choose  the  new  form: 


Ni  *  N.  (Xi  -  X.) 
- i  * - - 


Ni  +  Nj 


Xi  +  Xj 


(2) 


Here,  the  grayshade  values  in  the  denominator  tend  to  increase 
the  cost  when  the  grayshade  of  a  region  is  low  (close  to  zero) . 
The  goal  is  to  increase  the  cost  of  combining  a  dim,  cloudy 
region  with  a  dark,  background  region.  When  both  gray shades  are 
low,  the  grayshade  difference  in  the  numerator  will  be  small, 
thus  keeping  the  cost  low  for  combining  dark  regions. 

The  segmentation  was  rerun  using  the  cost  function  in  (2) 
(Fig.  5) .  Notice  that  there  is  very  little  difference  between 
this  segmentation  and  the  original  (Fig.  5a  &  b  vs.  Fig.  2a  &  b) . 
The  only  real  difference  is  that  the  thin  cloud  band  extending 
northward  from  the  stratocumulus  region  is  now  combined  with  the 
eastern  front  (area  B)  rather  than  with  the  background  region  A. 
Thus,  it  may  be  that  the  new  cost  function  is  having  the  desired 
effect,  but  not  to  the  extent  necessary. 

To  increase  the  effect  of  the  new  cost  function,  we  now 
square  the  grayshade  term  in  the  denominator: 


- i  * - - 


-  Xj) 


N,-  +  N- 


(3) 


The  ensuing  segmentation  is  presented  in  Fig.  6.  As  before,  the 
western  front  (area  +)  and  the  ITCZ  (area  8)  are  well-segmented. 
Notice,  however,  the  added  complexity  in  the  regions  describing 
the  background  (regions  A  and  C)  and  outer  space  (regions  1,  3,  6 
and  [)  areas.  By  squaring  the  denominator  of  the  grayshade  term. 
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Figure  5a.  As  in  Fig.  2a  except  for  HSWO  segmentation  using  cost 
f unct ion  in  Eqn .  ( 2 )  . 
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Figure  6a.  As  in  Fig.  2a  except  for  HSWO  segmentation  using  cost 
f unct ion  in  Eqn  ,  ( 3  )  . 


Figure  6b.  As  in  Fig.  2b  except  for  HSWO  segmentation  using  cost 
function  in  Eqn .  ( 3  )  . 
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the  cost  of  combining  dark  regions  is  now  so  high  that  these 
false  details  occur.  The  stratocumulus  region  is  still  included 
in  the  eastern  front  (area  2) ,  but  notice  that  more  of  the  thin, 
tailing  portion  is  now  included  as  well.  Thus,  the  new  cost 
function  is  beginning  to  have  the  desired  effect.  It  may  seem 
that  the  anomalous  detail  in  the  dark  regions  would  be  a  problem. 
However,  we  are  only  interested  in  segmenting  the  cloudy  areas 
well  so  that  they  can  be  classified  later.  Thus,  in  the  actual 
use  of  these  segmentations  one  could  simply  refer  to  the  average 
grayshade  of  each  region  and  discard  those  that  are  too  dark  to 
be  cloud  features. 

The  segmentation  still  fails  to  incorporate  the  entire 
eastern  front.  We  can  increase  the  effect  of  the  grayshade  term 
in  the  denominator  by  omitting  its  nonlinear  portion.  In  other 
words,  rather  than  squaring  the  sum  of  the  two  grayshades,  simply 
square  each  individual  grayshade: 

Ni  *  N.  (Xi  - 

Ci  j  - - 1  *  - - i--  (4) 

+  Nj 

This  final  segmentation  is  presented  in  Fig.  7.  The  western 
front  (A)  now  includes  some  of  the  broken  cloudiness  south  of  its 
tail,  but  the  overall  shape  is  still  correct.  The  ITCZ  (area  ') 
still  spans  the  image  nicely.  The  broken  cloudiness  north  of  the 
ITCZ  is  assigned  its  own  region  (area  9)  and  the  outer  space 
regions  continue  to  have  anomalous  detail  (areas  1,  3,  *  and  +) . 
The  most  encouraging  aspect  of  this  segmentation  is  its  ability 
to  include  the  entire  eastern  front  (area  C) .  The  region  contin¬ 
ues  to  include  the  stratocumulus  area  as  well.  This  error  seems 
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Figure  '’b.  As  in  Fig.  2b  except  for  HSWO  segmentation  using  cost 
function  in  Eqn  .  ( 4  )  . 
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unavoidable  because  the  connecting  cloud  band  is  so  bright.  It 
would  be  difficult  to  come  up  with  a  scheme  that  separates  these 
two  regions  while  also  connecting  the  eastern  front  with  its 
tail . 

There  are  likely  many  such  problem  cases  that  arise  in  such 
cloud  scenes.  To  come  up  with  a  meaningful  segmentation  would 
seem  to  require  some  top-level  information  to  resolve  such  prob¬ 
lem  areas.  Regions,  such  as  the  western  front,  that  fit  the 
classical  model  of  such  features  and  are  not  connected  or  adja¬ 
cent  to  other  cloudy  areas  can  be  easily  segmented.  However, 
problem  regions  such  as  area  C  (Fig.  7)  would  require  some  post¬ 
processing  to  determine  that  the  stratocumulus  area  should  be 
separated  from  the  frontal  band. 

Finally,  the  stopping  criterion  should  be  examined  in  these 
cases.  Figures  2c  -  7c  reveal  that  the  cloud  imagery  exhibits  a 
similar  minimum  cost  function  jump  as  that  demonstrated  by  Beu- 
lieu  and  Goldberg  (1989) .  The  choice  of  stopping  point  based  on 
these  curves  is  fairly  straightforward.  A  closer  examination  of 
the  errors  in  the  various  segmentations  reveals  that  the  particu¬ 
lar  choice  of  stopping  point  is  usually  not  the  cause  of  the 
errors.  Combinations  that  should  not  have  occurred  typically 
happen  several  iterations  prior  to  the  selected  stopping  point. 
Similarly,  combinations  that  should  have  occurred  would  not  have 
happened  until  many  iterations  after  the  stopping  points.  There 
fore,  the  important  issue  seems  to  be  the  HSWO  scheme  itself 
(e.g. ,  the  choice  of  cost  function)  rather  than  the  stopping 
point. 
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4. 


Case  Studies 


In  this  section  six  case  studies  of  the  HSWO  routine  will  be 
presented.  The  images  chosen  are  taken  every  three  days  from 
Oct.  1  -  Oct.  16,  1983.  As  a  basis  for  comparison,  the  original 
HSWO  segmentations  are  presented  along  with  the  segmentations 
that  result  from  the  use  of  the  new  cost  function  in  Eqn.  (4) . 

As  will  be  shown,  these  cases  contain  very  complex  cloud  regions 
rather  than  the  nice,  isolated  features  such  as  those  seen  in 
the  Nov.  15  case. 

4.1  October  1,  1983  Case 

This  case  is  characterized  by  a  large,  frontal  band  extend¬ 
ing  from  the  Bering  Sea  to  the  southwest  edge  of  the  image  (Fig. 
8) .  There  is  a  cloud  vortex  west  of  Hawaii  that  indicates  a 
cutoff  low.  The  area  from  Hawaii  to  the  U.S.  west  coast  is 
characterized  by  stratocumulus  cells.  A  frontal  band  extends 
from  the  Rockies  southwest  to  the  ocean  west  of  Baja.  The  ITCZ 
spans  the  tropics  and  includes  a  tropical  cyclone  in  the  east 
Pacific. 

The  segmentation  using  the  original  HSWO  scheme  is  presented 
in  Fig.  9.  The  western  front  has  been  divided  into  two,  adjacent 
regions  (areas  6  &  12) .  The  cloud  vortex  has  been  well-defined 
in  region  9.  The  stratocumulus  area  is  absorbed  into  background 
areas  2  and  3.  The  eastern  front  (area  5)  is  handled  well. 

There  is  a  lot  of  fine  detail  included  in  the  ITCZ  (areas  4,  8, 
10,  11,  15,  19  and  22)  although  such  detail  brings  out  the  tropi¬ 
cal  cyclone  (area  7) .  The  stopping  criterion  for  this  case  (Fig. 
9c)  is  based  on  a  relatively  small  jump  at  15  regions.  If  the 
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Figure  9a.  As  in  Fig.  2a  except  for  original  HSWO  segmentation 
using  image  in  Fig.  8. 


Figure  9b.  As  in  Fig.  2b  except  for  original  HSWO  segmentation 
using  image  in  Fig.  8. 


Figure  9c.  As  in  Fig,  2c  except  for  original  HSWO  segmentation 
using  image  in  Fig,  8. 
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jump  at  11  regions  had  been  used,  much  of  the  ITCZ  detail  would 
have  been  combined. 

When  using  the  new  segmentation  based  on  Eqn.  (4),  the  image 
again  contains  much  more  detail  (Fig.  10).  The  western  front 
(area  8)  is  now  one  complete  region.  It  is  curious  that  the 
southmost  portion  of  the  front  is  still  separated,  and  is,  in 
fact,  combined  with  the  clouds  associated  with  the  cutoff  low. 

For  this  feature,  the  original  HSWO  appears  superior.  Neither 
scheme  handles  the  stratocumulus  region  well.  The  eastern  front 
is  well-handled  by  both  schemes.  One  advantage  in  the  new  scheme 
is  that  the  ITCZ  is  resolved  as  one,  large  band.  The  tropical 
cyclone,  however,  is  not  resolved  by  this  method.  As  stated  in 
the  previous  section,  the  anomalous  detail  in  the  dark  regions 
can  be  easily  handled  in  post-processing. 

Neither  scheme  is  the  clear  winner  on  this  case.  It  is 
important  that  the  western  front  be  resolved  as  one  band,  but  the 
failure  to  isolate  the  cutoff  low  by  the  new  scheme  is  disap¬ 
pointing. 

4.2  Oct.  4,  1983  Case 

This  case  (Fig.  11)  has  a  very  complex  cloud  region  in  the 
Bering  Sea.  R.  Fett  identified  a  nearly  east-west  oriented 
frontal  band  with  open  cells  to  the  north  and  stratocumulus  to 
the  south.  He  also  identified  the  bright  clouds  at  the  tip  of 
the  Alleutians  as  another  frontal  band.  The  bright  area  west  of 
California  is  dense  fog.  The  same  frontal  band  seen  on  the  Oct. 

1  image  over  Baja  is  seen  here  (this  may  not  be  a  front  after 
all),  along  with  the  tropical  cyclone  now  to  the  north  of  the 
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Figure  10a.  As  in  Fig.  2a  except  for  HSWO  segmentation  on  image 
in  Fig.  8  using  cost  function  in  Eqn .  (4). 


Figure  10b.  As  in  Fig.  2b  except  for  HSWO  segmentation  on  image 
in  Fig.  8  using  cost  function  in  Eqn.  (4). 
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Figure  10c.  As  in  Fig.  2c  except  for  HSWO  segmentation  on  image 
in  Fig.  8  using  cost  function  in  Eqn.  (4). 
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ITCZ .  The  ITCZ  is  most  active  in  the  west,  with  a  lear  break 
south  of  the  tropical  cyclone. 

The  HSWO  (Fig.  12)  isolates  the  Bering  cold  fronts  into  a 
single  region  (area  7).  This  region  also  includes  some  of  the 
open  cells  to  the  north  and  the  fog  region.  The  stratocumulus 
region  is  absorbed  into  background  area  4.  The  Baja  cloud  band 
is  well-defined  (area  5).  There  is  much  structure  in  the  ITCZ 
here  as  well. 

The  modified  HSWO  (Fig.  13)  handles  the  complex  Bering 
fronts  in  a  similar  fashion.  The  Baja  front  (area  ^)  is  also 
well-handled.  There  is  not  quite  as  much  complexity  in  the  ITCZ, 
although  the  central  portion  has  been  combined  into  the  back¬ 
ground  area  A. 

Again,  there  is  no  clear  superiority  of  either  scheme.  This 
image  contains  some  very  complex  features  that  are  difficult  for 
even  an  expert  to  segment  correctly  without  using  some  top-level 
knowledge  of  what  the  features  might  be. 

4.3  Oct.  7,  1983  Case 

The  cloud  band  in  the  extreme  west  (Fig.  14)  is  a  meridional 
trough.  There  is  a  front  in  the  Bering  Sea  with  a  vortex  to  the 
south  and  a  thin,  dissipating  front  extending  toward  Hawaii. 

There  is  dense  fog  off  the  northern  California  coast  and  the 
remnants  of  the  tropical  cyclone  over  southern  California. 

The  segmentation  (Fig.  15)  again  handles  the  complex  Bering 
sea  features  by  combining  them  into  a  single  region  B.  The 
dissipating  front  is  completely  lost.  The  meridional  trough 
appears  as  two,  separate  regions  D  and  F.  The  tropical  cyclone 
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Figure  12a.  As  in  Fig. 
using  image  in  Fig.  11. 


2a  except  for  original  HSWO  segmentation 


Figure  12b.  As  in  Fig. 
using  image  in  Fig.  11. 


2b  except  for  original  HSWO  segmentation 
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Figure  15a.  As  in  Fig.  2a  except  for  original  HSWO  segmentation 
using  image  in  Fig.  14. 


Figure  15b.  As  in  Fig.  2b  except  for  original  HSWO  segmentation 
using  image  in  Fig.  14. 
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Figure  15c.  As  in  Fig.  2c  except  for  original  HSWO  segmentation 
using  image  in  Fig.  14. 
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is  handled  well  (area  @)  and  there  is  the  typical  overstructuring 
of  the  ITCZ  (areas  4,  9,  A  and  G). 

The  modified  HSWO  segmentation  (Fig.  16)  also  has  difficulty 
with  the  Bering  Sea  features ,  but  makes  matters  even  worse  by 
extending  the  region  (area  @)  across  the  western  U.S.  The  meri¬ 
dional  trough  is  combined  with  the  dim  clouds  to  the  west  (area 
-).  Even  the  tropical  cyclone  is  not  resolved  here.  The  one 
good  point  is  that  the  ITCZ  is  resolved  as  a  single,  continuous 
band. 

The  original  segmentation  seems  to  be  better  here  if  one  is 
interested  in  resolving  the  mid-latitude  features,  although 
considerable  post-processing  would  still  be  required.  The  modi¬ 
fied  HSWO  seems  to  handle  the  ITCZ  better,  in  general. 

4.4  Oct.  10,  1983  Case. 

This  case  (Fig.  17)  finally  departs  from  the  very  complex 
features  to  a  more  "normal"  scene.  There  is  a  frontal  band 
extending  from  Alaska  to  the  southwest.  Ahead  of  the  front  is  a 
vortex  associated  with  an  upper  cold  low.  West  of  Baja  is  a 
stratocumulus  region  and  a  hurricane  to  the  south. 

Unlike  the  other  cases,  the  cost  function  curve  of  this  case 
(Fig.  18c)  had  no  good  stopping  point  short  of  only  six  regions. 
In  that  case  (not  shown),  only  the  frontal  band  is  resolved,  and 
the  remaining  areas  are  background  regions.  If  we  accept  a  35% 
rise  in  cost  as  a  cutoff,  one  can  stop  at  19  regions  (Fig.  18c). 
This  situation  is  shown  in  Fig.  18a  &  b.  Here  the  front  is 
resolved  well  (area  5).  The  vortex  is  partially  resolved,  but  it 
is  also  included  into  the  cloudy  region  over  the  northwest  U.S. 
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Figure  16a.  As  in  Fig.  2a  except 
in  Fig.  I4  using  cost  function  in 


for  HSWO  segmentation  on  image 
Eqn .  ( 4  )  . 


Figure  16b.  As  in  Fig.  2b  except  for  HSWo  segmentation  on 
in  Fig,  14  using  cost  function  in  Eqn .  (4) 
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Figure  16c.  As  in  Fig.  2c  except  for  HSWO  segmentation  on  image 
in  Fig.  14  using  cost  function  in  Eqn .  (4). 
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Figure  17.  As  in  Fig.  l  except  for  Oct.  10,  1983  image. 
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Figure  18a.  As  in  Fig.  2a  except  for  original  HSWO  segmentation 
using  image  in  Fig.  17. 


Figure  18b.  As  in  Fig.  2b  except  for  original  HSWO  segmentation 
using  image  in  Fig.  17. 
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[ure  18c.  As  in  Fig.  2c  except  for  original  HSWO  segmentation 
ng  image  in  Fig.  17. 
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(area  -).  The  stratocumulus  region  is  resolved  (area  6)  as  is 
the  hurricane  (area  A).  Again,  there  is  much  structure  assigned 
to  the  ITCZ  and  background  regions. 

The  modified  HSWO  (fig.  19)  also  resolves  the  front  well 
(area  -)  and  the  vortex  is  again  included  into  the  cloudy  region 
to  the  east  (area  <).  Here,  the  hurricane  and  stratocumulus 
regions  are  combined  (area  D)  along  with  a  portion  of  the  ITCZ. 
The  original  HSWO  is  clearly  superior  in  this  case. 

4.5  Oct.  13,  1983  Case. 

In  this  image  (Fig.  20)  there  is  a  broad,  frontal  band 
extending  from  the  Alleutians  to  the  southwest.  There  is  anoth¬ 
er,  thin  front  to  the  east  in  the  Bering  Sea.  There  is  an  area 
of  stratocumulus  off  the  northern  California  coast  and  another 
one  northwest  of  a  large  hurricane. 

The  segmentation  of  this  image  (Fig.  21)  captures  the  large 
front  well  (area  B),  but  the  northern  stratocumulus  area  is  also 
incorporated.  The  hurricane  (area  C)  is  well  handled,  but  the 
southern  stratocumulus  area  is  combined  into  background  area  A. 
Major  features  of  the  ITCZ  are  isolated  (areas  3,  7,  8,  +  and  !). 

The  modified  HSWO  (Fig.  22)  also  does  well  in  isolating  the 
broad  front  (area  *),  and  this  method  does  not  incorporate  the 
northern  stratocumulus  area.  Unfortunately,  that  area  is  ab¬ 
sorbed  into  background  region  5 .  The  hurricane  is  segmented 
into  area  <.  Here,  the  southern  stratocumulus  area  gets  ab¬ 
sorbed  into  the  western  ITCZ.  The  scheme  is,  however,  able  to 
isolate  the  southern  portion  of  the  thin  front  (area  +),  al¬ 
though  some  of  the  western  portion  of  this  region  is  background. 
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Figure  19a.  As  in  Fig.  2a  except  for  HSWO  segmentation  on  image 
in  Fig.  17  using  cost  function  in  Eqn.  (4). 


Figure  19b.  As  in  Fig.  2b  except  for  HSWO  segmentation  on  image 
in  Fig.  17  using  cost  function  in  Eqn.  (4). 
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Figure  21a.  As  in  Fig.  2a  except  for  original  HSWO  segmentation 
using  image  in  Fig.  20. 


Figure  21b.  As  in  Fig.  2b  except  for  original  HSWO  segmentation 
using  image  in  Fig.  20. 


51 


Figure  22a.  As  in  Fig.  2a  except  for  HSWO  segmentation  on  image 
in  Fig.  20  using  cost  function  in  Eqn .  (4). 


Figure  22b.  As  in  Fig.  2b  except  for  HSWO  segmentation  on  image 
in  Fig.  20  using  cost  function  in  Eqn.  (4). 
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4.6  Oct.  16,  1983  Case 

This,  final  case  (Fig.  23)  again  has  a  standard  front  in  the 
Bering  Sea.  There  is  an  area  of  stratocumulus  off  of  California 
and  Baja.  A  tropical  storm  is  located  to  the  west  as  well  as  a 
hurricane  off  the  Mexican  coast. 

The  segmentation  (Fig.  24)  fails  to  resolve  the  cold  front 
in  this  image  because  there  is  a  broad  region  of  bright,  strato¬ 
cumulus  cells  in  the  cold  air  behind  the  front.  The  tail  of  the 
front  gets  absorbed  into  this  region  (area  A) .  The  nature  of 
these  photographs  is  to  have  a  generally  brighter  reflection 
along  the  planetary  edges.  The  body  of  the  front  gets  absorbed 
into  this  anomalous  bright  region  (area  C)  that  extends  across 
the  top  of  the  image  and  even  down  into  the  tropics.  The  strato¬ 
cumulus  off  of  California  is  segmented  well  (area  ,  as  are  the 
tropical  cyclones  (areas  ‘  and  *) .  Only  a  small  part  of  the 
western  (area  8)  and  eastern  (area  $)  ITCZ  is  resolved. 

The  modified  HSWO  (Fig.  25)  falls  victim  to  the  same  circum¬ 
stances  regarding  the  front.  The  tail  of  the  front  combines  with 
background  area  D  and  the  head  with  the  edge-effect  area  C.  Even 
the  tropical  cyclones  are  absorbed  into  area  C.  The  western  ITCZ 
(area  =)  is  segmented  well,  but  the  eastern  ITCZ  is  lost. 

4.7  Discussion 

Upon  examination  of  these  six  cases,  the  HSWO  tests  on  the 
Nov.  15  case  appear  to  have  been  deceptively  good.  The  image  at 
hand  had  well-defined  regions  that  were  easily  distinguished 
from  the  background  and  from  each  other.  The  six  cases  present¬ 
ed  here  show  the  more  typical  nature  of  the  imagery  to  be  proc- 
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Figure  24c,  As  in  Fig.  2c  except  for  original  HSWO  segmentation 
using  image  in  Fig.  23. 
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Figure  25a.  As  in  Fig.  2a  except  for  HSWO  segmentation  on  image 
in  Fig.  23  using  cost  function  in  Eqn .  (4). 


Figure  25b.  As  in  Fig.  2b  except  for  HSWO  segmentation  on  image 
in  Fig.  23  using  cost  function  In  Eqn.  (4). 
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for  HSWO  segmentation  on  image 
Eqn.  (4). 


essed.  The  cloudy  areas  can  be  quite  complex,  and  it  takes 
considerable  skill  to  be  able  to  isolate  the  meaningful  regions. 
HSWO  alone  is  not  capable  of  doing  this  task. 

It  may  be  possible  to  improve  the  HSWO  segmentations  by 
post-processing  the  initial  regions.  Perhaps  a  rule-based 
system  or  a  neural  net  could  specify  which  regions  are  correct¬ 
ly-segmented  cloud  features  and  which  need  to  be  segmented 
further  or  combined  together. 

Alternately,  a  different  segmentation  approach  might  be 
devised  which  applies  constraints  on  the  region-growing  process. 
In  other  words,  as  the  areas  are  considered  for  combination,  the 
method  would  consider  whether  the  resulting  region  makes  sense  in 
terms  of  its  shape  and  its  context  in  the  emerging  scene.  Such  a 
methodology  would  require  fairly  basic  research  in  image  segmen¬ 
tation  theory.  However,  it  seems  clear  that  the  meteorological 
features  that  must  be  isolated  depend  on  more  than  simple  bright 
vs.  dark  considerations.  In  particular,  it  would  be  interesting 
to  apply  the  so-called  Boltzmann  machine  neural  network  methodol¬ 
ogy  to  this  problem. 

The  experiments  conducted  here  have  been  conducted  on  digi¬ 
tized,  photographic  images.  It  may  be  that  the  use  of  original 
satellite  data  would  produce  better  results.  NOARL  has  a  number 
of  mosaiced  DMSP  images  that  might  be  tried,  as  well  as  some 
fairly  wide-view  AVHRR  data.  It  may  be  useful  to  try  HSWO  on 
these  cases. 

There  are  other  region-growing  methods  in  the  literature 
besides  HSWO.  It  may  be  illuminating  to  conduct  a  literature 
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search  of  other  image  segmentation  methods. 

Finally,  it  may  be  that  the  segmentation  is  such  an  integral 
part  of  the  feature  classification  process  that  it  simply  cannot 
be  done  as  a  single  step.  Perhaps  one  could  isolate  the  easily- 
discernible  features,  and  then  proceed  as  does  the  SIAMES  expert 
system.  Using  constraints  and  partial  knowledge,  one  could  begin 
an  intelligent  search  of  the  remaining  image  for  other  features 
of  interest.  Such  a  top-down  approach  combined  with  partial, 
bottom-up  information  might  work. 

5.  Conclusions 

In  this  paper,  the  Hierarchical  Stepwise  Optimization 
algorithm  is  applied  to  the  problem  of  segmenting  cloud  features 
in  satellite  images.  Various  forms  of  the  cost  function  have 
been  explored  as  potential  ways  to  improve  the  segmentations. 

In  tests  on  the  Nov.  15,  1983  image,  the  form  given  in  Eqn.  (4) 
proved  to  be  superior. 

The  original  HSWO  and  the  modified  version  have  been  tested 
on  six  images.  Although  both  forms  of  the  algorithm  show  prom¬ 
ise  in  the  ability  to  segment  certain  cloud  features,  the  scheme 
is  quite  inadequate  in  discriminating  many  of  the  complex  cloud 
patterns  that  appear  in  these  images.  A  certain  amount  of  the 
problem  may  result  from  the  use  of  data  digitized  from  photo¬ 
graphs.  However,  it  appears  likely  that  the  very  nature  of  the 
types  of  cloud  features  of  interest  would  thwart  attempts  to 
provide  a  segmentation  based  only  on  bright  vs.  dark  considera¬ 
tions. 

It  is  recommended  that  a  new  approach  be  devised  that  takes 
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into  account  some  top-level  information  about  large-scale  cloud 
features.  Constraints  on  the  size  and  shape  of  features  al¬ 
lowed,  and  on  the  spatial  interrelations  of  such  features, 
should  be  included  in  the  segmentation  scheme.  For  example,  we 
know  that  the  ITCZ  has  a  preferred  location  in  the  tropics  and 
that  it  tends  to  span  the  width  of  the  image.  Similarly,  fron¬ 
tal  bands  tend  to  be  located  and  oriented  in  certain  ways. 
Segmentation  into  the  more  likely  cloud  configurations  would  be 
given  preference  over  segmentations  that  do  not  fit  the  expected 
model  as  well. 

The  neural  network  methodology  known  as  the  "Boltzman  Ma¬ 
chine"  is  a  constraint-satisfaction  methodology.  These  networks 
are  useful  in  solving  problems  that  must  satisfy  many  con¬ 
straints,  some  of  which  may  even  be  contradictory.  This  method¬ 
ology  should  be  explored  to  see  whether  the  segmentation  problem 
could  be  defined  in  a  form  that  it  could  solve.  Additionally, 
the  literature  should  be  scanned  to  see  whether  there  are  any 
other  approaches  that  may  be  applicable  to  the  cloud  feature 
segmentation  problem. 
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